HermitCrab optimization round record: what was tried, and a faster package that is not mergeable - #490

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.57%. Comparing base (74ee09b) to head (2eba2f5).
⚠️ Report is 1 commits behind head on master.

@@           Coverage Diff           @@
##           master     #490   +/-   ##
=======================================
  Coverage   74.57%   74.57%           
=======================================
  Files         453      453           
  Lines       38314    38314           
  Branches     5280     5280           
=======================================
  Hits        28573    28573           
  Misses       8576     8576           
  Partials     1165     1165           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Two further optimization ideas were investigated against this ledger, on a branch based on master (a4b2974). Both are closed. Recording them here because the ledger's purpose is to stop the work being redone, and one of them looks irresistible on paper.

Neither appears in the existing 22 rows. Row 5 covers deterministic synthesis fold sharing and the rejected two-pass experiment covers nondeterministic traversal; neither is the idea below. Row 6 is the synthesis-side sibling of the second idea.

Idea A — a traversed dedup set for the deterministic FSA traversal

NondeterministicFsaTraversalMethod keeps a traversed set keyed on (State, AnnotationIndex, Registers); DeterministicFsaTraversalMethod does not. The motivating observation was that a deterministic FST can still be traversed nondeterministically when the input is nondeterministic — Annotation.Optional makes Advance fork skip-vs-consume, and the forks reconverge.

The mechanism is real and the key is complete on that path (no output operations, no priorities, and Matcher.Compile only determinizes when there are no variables, so VariableBindings is always null). On a synthetic input with 8 consecutive optional annotations it takes instance pushes from 510 to 44 with identical matches.

It does not pay on real grammars. Five FieldWorks exports, tracing on, each word parsed twice (dedup off then on), one word per process, no parse timeout:

Parse-signature multisets identical on every completed word. The reason it is a no-op is structural: the deterministic traversal only runs on the synthesis side (analysis affix matchers set AllSubmatches, analysis rewrite/metathesis set Nondeterministic), and synthesis shapes carry almost no optional annotations, so the skip-vs-consume fork it would collapse rarely occurs. Deterministic traversal is 0.02–3.4 % of all CheckInputMatch calls, and dedup removes ≤ 0.8 % of that.

Where the duplicate explosion the idea was aiming at actually lives is the nondeterministic set, which already exists: on Amharic it suppressed 30.1 M candidate instances against 15.2 M pushed (66 % duplicates), and on one heavy word 82.7 M against 38.5 M.

Two latent issues noticed while auditing the existing keys, neither reachable from HermitCrab today:

• NondeterministicFstTraversalMethod's key carries the output action list but not which annotations the actions applied to. Two paths through optional annotations can share state, index, registers and action sequence while having modified different output annotations. Unreachable because no HermitCrab FST carries IFstOperations.
• Both nondeterministic keys omit VariableBindings, which CheckInputMatch reads. Whether two equal (State, AnnotationIndex, Registers) keys can carry different bindings on a real pattern is untested.
• Both store the mutable Register[,] directly in the key while ExecuteCommands later mutates it. This can only cause a missed dedup, never a wrong one.

Idea B — canonicalize the order of independent rules in an Unordered analysis stratum

The analysis-side counterpart of row 6. In an Unordered stratum the cascade explores both [A,B] and [B,A]; if A and B commute, one order is redundant work. The prune skips a dispatch when the candidate is orthogonal to the trail's last rule and precedes it in source order — adjacent-transposition normalization, with non-orthogonal rules and templates acting as barriers.

Orthogonality was implemented as three cumulative levels — L0 shape (pure prefix vs pure suffix, single application, edge-blind stem pattern, no ModifyFromInput/variables), L1 + syntactic-feature commutation decided by enumerating the finite feature lattice the two rules mention in both the analysis (IsUnifiable gate then Add/Clear) and synthesis (Unify then PriorityUnion) directions, L2 + MPR/stem-name/partial/blocking. A second, independently written census over the XML agreed with the object-model implementation on every unordered stratum: 176 L0 pairs, 60 L1, 36 L2 across the five grammars.

Measured: 0.00–0.05 % of rule-unapplication attempts, parity clean on every word.

The L0 control is what makes this a finding rather than a null result: widening Sena from 2 qualifying pairs to 52 moves the reduction from 0.05 % to 6.37 %, so the limit is the predicate, not the hook. But each narrowing step is load-bearing — of Sena's 52 L0 pairs, 30 fail L1 because their syntactic features genuinely do not commute, and 20 fail L2 on the partial-rule clause, which is necessary there because that stratum holds 24 affix templates so IsLastAppliedRuleFinal takes a value and both final-template gates in SynthesisAffixProcessRule can fire. 6.37 % is what an incorrect prune wins.

A trail-depth census explains why widening cannot reach further. Instrumenting every dispatch with the length of the trail it extends:

• Mbugwe, 155,672 dispatches: 68 % sit on trails 9+ rules deep, but at only 0.8 % does the trail's last rule have any orthogonal partner even at L0.
• A heavy Sena word, 76,777 dispatches: 67 % at depth 9+, 16.6 % partner-available, 2.3 % actually skipped → the 6.37 %.

So the permutation blow-up is real and deep, but the rules filling those trails are ones the criterion excludes on shape (Mbugwe rejects 57 pairs MultiInsert, 17 AllomorphOutput, 21 RuleKind). The largest excluded class is same-side pairs (Sena: 84 rejections, the C(13,2) suffix pairs), and those can never commute — stem+A+B ≠ stem+B+A is arithmetic, not conservatism.

One consequence worth separating from performance. The prune is not signature-preserving and cannot be: a parse signature carries MorphemesInApplicationOrder, so unapplying a prefix before a suffix and the reverse are two distinct parses of one word even when the rules commute. What it preserves is the morph sequence and the root; what it drops is derivational history. Those two rows are indistinguishable to a reader, so this is an ambiguity reduction as much as an optimization — and it would need the correctness treatment rather than being presented as invisible.

This is now the fifth instance of the pattern in this ledger's "lesson that generalises": apparent redundancy that evaporates once the key is complete.

On rows 10 and 11 — reachability

The first reading of our data looked like it reopened the lexical reachability gate: the heavy Sena word made 272,152 unapplication attempts and the Aweti word 505,600, each finding zero roots. It does not reopen it. Those lexical-lookup counts come from runs truncated by a step budget, so they show the search never reached lookup within budget, not that lookup would have failed. Row 11's finding — that the expensive words fail on checks running after lexical lookup succeeds — is untouched by anything measured here.

On the complexity-cap step budget

b3fd2b55 was cherry-picked to bound the pathological words. It works where the cost is many rule applications: an Aweti word that previously crashed a test host at 12.1 GB after 1,717 s completes in 50 s at 200,000 steps, and whole-corpus runs go from 22/40 words to 40/40 at 757 MB peak instead of 12 GB.

It does not bound a word whose cost is one expensive traversal. An Amharic word ran 2,280 s under a 200,000-step budget with the nondeterministic counters frozen at 217,493 traversals and 53.9 M arc checks across successive samples — stuck inside a single matcher call. The budget is checked between leaf-rule applications and cannot interrupt one. Anyone relying on it as a safety bound should know that.

Also worth flagging for consumers: the budget ships on by default (2,000,000 steps, 10 s). A 10-second default silently truncates parses.

Reproduction

Branch based on master a4b2974. Measurements used a census harness that parses each word twice in one process with MaxDegreeOfParallelism: 1 and tracing on (so the analysis memo and MergeEquivalentAnalyses are off and counts are the unmemoized search), compares complete parse-signature multisets, and reports MorphologicalRuleAnalysis trace nodes as attempts and CheckInputMatch calls as arc checks. Grammars are FieldWorks GenerateHCConfig exports of five projects; word lists are the first 40 entries except Indonesian (all 121) and Sena (entries 1001–1040, since the head of that list is punctuation). Machine-safety aborts and test-host crashes are recorded as such and excluded from ratios, never counted as agreement.

Two observations from the same investigation that belong in the record but are not optimizations, so they did not fit the rows above.

A repeatable memory wall at 12.2–12.6 GB, independent of any cap

Across the five-grammar runs, 13 words (Sena and Aweti) ended as test-host crashes rather than completions, and they did so at a strikingly consistent managed heap: 12,225 / 12,259 / 12,270 / 12,278 / 12,282 / 12,312 / 12,323 / 12,324 / 12,336 / 12,340 / 12,359 / 12,365 / 12,563 MB. The consistency holds regardless of the per-process limit in force — including runs whose limit was 20 GB, so these are not the harness stopping them.

The clustering at ~12.3 GB rather than at whatever ceiling was configured is the interesting part; it is the signature one would expect from a single collection outgrowing the 2 GB per-object limit rather than from address-space or RAM exhaustion. A crash dump on one of these words would name the collection. Reported as an observation only — no diagnosis attempted.

For scale, one Aweti word (line 183 of its list) reached 13.2 GB and 30.1 M nondeterministic traversals / 139.3 M arc checks in 53 minutes without leaving analysis, and never reported a lexical lookup. Under a 200,000-step budget the same word completes in 231 s at 676 MB with 505,600 rule-unapplication attempts and 0 deterministic traversals.

Where analysis time goes, by traversal family

The counters added for this work split every CheckInputMatch call between the two FSA traversal families. Deterministic traversal — which is the synthesis side only, since analysis affix matchers set AllSubmatches and analysis rewrite/metathesis set Nondeterministic — accounts for:

This is offered as calibration for anyone sizing a synthesis-side optimization: on the two grammars with the heaviest analysis, 99.8 % or more of FST work is on the nondeterministic side. It is consistent with the ledger's Amharic breakdown (analysis cascade ~95 % of analysis, all synthesis 0.3 %), measured independently and at the arc-check level rather than by wall-time bucket.

Added one record from a September round that post-dates this set: docs/rejected-optimizations/state-position-traversal-dedup.md, covering PR #511's (State, AnnotationIndex) traversal dedupe.

Three things it contributes to this ledger:

1. A reproduction of the motivating pathology, with the mechanism named. It is not the state count — TraversalMethodBase.Advance forks an instance per Optional annotation, so instances are exponential in optional count and independent of |States|. Measured 2,097,150 instances popped on a 2-state FSA at 20 optional annotations.
2. A fourth row for "the lesson that generalises." Apparent collapse 93.95%, sound collapse 0.28%. Same shape as rows 5, 6 and the fold-entry census: a key that omits what makes instances distinct shows large shareable work that is not there. Here the omitted state is variable bindings (which filter later arcs) and registers (which carry the captures AllSubmatches callers consume).
3. A narrowed form that does pay, unlike most of this ledger: restricted to the deterministic method with no capture groups, the differential fuzz goes 112 divergences → 0 while the group-free microbenchmark stays cell-for-cell identical to the unnarrowed change.

Also closes the two extensions before anyone builds them — environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe (this is #515), and the analysis rules' 93.95% is 99.7% alternate captures because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries deliberately.

docs/rejected-optimizations/two-pass-nondeterministic-traversal.md now cross-references it: that experiment kept bindings in the key and rebuilt captures and was too expensive; this one drops both and is unsound. Opposite errors, same boundary.

🤖 Generated with Claude Code

2026-09-23 round: the missing Aweti speedup was reduplication, not traversal

One new row closes the gap #511 opened. Aweti index 182 had never completed here, on any stack we tried (#491, #494, #511, FieldWorks #1148, forced final templates, FLEx's parallel/no-memo mode). A copy-agreement prune in reduplication unapplication makes it finish in 9 s, and the whole Aweti list runs 4.95× faster with 0 divergences across five grammars. Proposed as #519 (on by default) and shipped in PanGloss v0.4.0.

New row

Why the ledger missed it: every prior row measured Sena, Mbugwe and Amharic, where full-copy rules are absent or rare. The 19 docs mention reduplication once (the probe-design classifier, marked "Unknown").

What else this round settled

• The Aweti blow-up is 12 full-copy rules (mrule25-mrule36), not traversal. Analysis captures each copy independently and uses only the first, so every split becomes an analysis. Across the list, 99.7% of the matches that can be judged disagree.
• Add allMatches to Traverse function to improve performance #511 as reviewed was unsound and gave no gain on our grammars. The old key produced 112/20k fuzz divergences; a narrowed form was fuzz-clean but no faster. See "Open since this round" for the redesign.
• #1148 changes 1 entry / 1 allomorph on Aweti and Sena, and no speed. It can also drop a co-occurrence prohibition that references a removed duplicate MSA (reproduced).
• FLEx never runs the Memoize HermitCrab's sequential analysis cascade #456 memo. FieldWorks (HCParser.cs:178) and the HC tool construct Morpher with the default parallelism, and the memo only engages at degree 1. On 21 completing Aweti words, the default mode uses 2.93× the CPU of the memo mode, with identical analyses. That makes it a separate FieldWorks change worth measuring.
• Master is not worse than Maxwell's build. FLEx's own mode (parallel, no memo) also OOMs on index 182. His reported 83 s → 3 s and 4 → 1 min match this prune's effect; how his Add allMatches to Traverse function to improve performance #511/#1148 stack got there is still unexplained.
• Closed, don't revisit:
  • Root-length pre-lookup check: rejects 1.46% of candidates.
  • Traversal dedup and extending it to allMatches: 0.31% of pushes collapsible.
  • Env-matcher dedup (Environment matchers pay for captures they never read; key the traversal on what the caller consumes #515).
  • Row 16 on Aweti: 15.45% duplicates on a 177-candidate prefix, inside the fail band but unfinished.
• Open, research-only:
  • Lazy composed FST for the phonology (unmeasured).
  • Two-way FSTs for productive copying: literature only, no HC mapping yet.

PanGloss v0.4.0 (same prune, Rust)

Release binaries, same machine, v0.3.3 vs v0.4.0: identical analyses on all 7,455 words both complete across the five grammars. Aweti: 172 -> 191 words complete within 20 s, 5.3x on the 172 shared words, median word 188 -> 26 ms, word-list index 182 60 s timeout -> 2.7 s. Sena, Amharic and Mbugwe are within noise to about 10% slower; a per-call allocation on rules that copy nothing is being removed.

Open since this round

• Add allMatches to Traverse function to improve performance #511 was redesigned on 2026-09-24 (variable bindings in the key, a lattice on the nondeterministic path). The fuzz above ran on the old key; the new head is being re-fuzzed now.
• Remove HermitCrab's analysis-cascade memo #522 removes the C# analysis memo, matching PanGloss (which removed it earlier).

Method notes worth keeping

• Parity used gloss/allomorph/index signatures. Morpheme.Id is empty in FLEx exports.
• The fixture edge-cases-zero-width-morpheme-identity-stability is nondeterministic on master (off/off hashes differ); exclude it from any off/on diff.
• A seeded fuzz that compares "baseline minus classified" against "pruned" only tests the mechanism, not soundness. Soundness came from conformance, the parameterized ReduplicationRules/ModifyFromInputRules tests, three mutants, and the real-grammar sweep.

🤖 Generated with Claude Code

1. Verdict: approve with fixes. This PR changes documentation only. The new commit records the R1 purity extension (PR perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579, ledger row 48). Its description of Untruncate matches master, and its arithmetic checks out. Its baseline for Require slot literals in order and at reachable edges in the template look-ahead #576 does not match the R3 record, though. F20-F44 have had no reply, and their lines are unchanged.
2. Most important: F40 (docs/hermitcrab-optimization-ledger.md:175) is still open: the R1-on-perf(hermitcrab): share trailing analysis template slot batches #546 KEEP never evaluates its own overlap rule. F33 (docs/hermitcrab-2026-10-27-analysis-round.md:54) is next: the two root censuses still contradict each other. New: F45 (docs/hermitcrab-2026-10-27-analysis-round.md:402). Require slot literals in order and at reachable edges in the template look-ahead #576 on Mbugwe is 8.19% less here, but the R3 section and row 44 give 7.3% less.
3. Counts: Critical 0. Important: 5 open (F20, F21, F26, F33, F40). Low: 22 open (F22-F25, F27-F32, F34-F39, F41-F44) and 2 new (F45, F46).
4. What I ran:
   • The compare 4b2f7557...2eba2f50 reports ahead: 1 commit, which adds the perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579 section, lesson 8 and ledger row 48. The diff was small, so I reviewed it by hand and did not run /code-review.
   • I checked AnalysisMorphologicalTransform.Untruncate (AnalysisMorphologicalTransform.cs:228-256). It adds one node for a segment constraint and loops MaxOccur times for a quantifier, so max=-1 restores nothing, as the record says.
   • I recomputed 40.85 - 8.19 = 32.66 points and 1,133,568 / 995,434 = 13.88% more. Both match.
   • gh pr checks 490: changes, Build on ubuntu-22.04, and both comment-hygiene checks pass. Build on windows-latest and claude-review are pending.
5. Not verified: the measured figures themselves, PRs Prune analysis rule outputs that contain no root when no later rule can restore stem material #572 and Skip impossible analysis affix matches using edge segments #574-perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579, PanGloss Unable to configure translation engines properly #67, and the 16% Aweti limit. They rest on branches and grammars I could not access.

Public API, target framework, package, and machine.py parity: None verified (docs only).

Findings: F1-F19 addressed; F20, F21, F22, F23, F24, F25, F26, F27, F28, F29, F30, F31, F32, F33, F34, F35, F36, F37, F38, F39, F40, F41, F42, F44 open; F43 open, unverified; F45, F46 new.

Reviewed at 2eba2f5

Closing: this record is now one issue per idea, labelled performance. Each issue links its full record at this branch's final commit (2eba2f5), and the branch stays.

• Still open: HermitCrab analysis: defer the per-template clone in AnalysisAffixTemplateRule (1.054x wall-clock, not a step count) #586 HermitCrab feature structures: skip the clone map for frozen flat trees (no isolated measurement) #587 HermitCrab feature structures: allocation-free ValueEquals for frozen flat trees (no isolated measurement) #588 HermitCrab feature structures: reuse traversal state in ValueEquals and Freeze (pooled per-thread sets, no isolated measurement) #589 HermitCrab FST: use flat register arrays in the traversal (CreateInstanceMDArray 11.5% to 0.2% of profile, wall-clock only) #591 HermitCrab shapes: build the node map inside Shape.CopyTo's copy pass (no isolated measurement) #592 HermitCrab engine: share frozen Shapes on internal Word clones (clone 36% to 9% of profile; peak memory 11 GB to 6 GB) #593 HermitCrab engine: share frozen syntactic feature structures on internal Word clones (about 15% of a heavy Sena profile, never isolated) #595, plus the edge prefilter in HermitCrab analysis: port the edge-segment prefilter (time-only, 0 steps) #564 (PR Skip impossible analysis affix matches using edge segments #574)
• Rejected, closed with reasons: HermitCrab synthesis: memoize and hoist Word.ExpandAlternatives (rejected: word clones unchanged at 4,747,013) #584 HermitCrab analysis: query lexical reachability before materializing compounding splits (rejected: ceiling about 2% of clones, 9% of traversals) #585 HermitCrab analysis: final-template interleaving prune with a grammar-wide partiality guard (rejected: zero prunes on Sena, Amharic, Mbugwe) #590 HermitCrab FST: deduplicate traversal instances by (State, AnnotationIndex) (rejected in general form: 112 of 20,000 fuzz cases differ) #594 HermitCrab FST: two-pass nondeterministic traversal (rejected: 2.3x traversal instances, arc checks down only 1.6%) #596